Applying RE-AIM to examine the impact of an implementation facilitation package to scale up a program for Veterans with Chronic Obstructive Pulmonary Disease

Background U.S. Veterans are four-times more likely to be diagnosed with Chronic Obstructive Pulmonary Disease (COPD) compared to the civilian population with no care model that consistently improves Veteran outcomes when scaled. COPD Coordinated Access to Reduce Exacerbations (CARE) is a care bundle intended to improve the delivery of evidence-based practices to Veterans. To address challenges to scaling this program in the Veterans’ Health Administration (VA), the COPD CARE Academy (Academy), an implementation facilitation package comprised of four implementation strategies was designed and implemented. Methods This evaluation utilized a mixed-methods approach to assess the impact of the Academy’s implementation strategies on the RE-AIM framework implementation outcomes and the extent to which they were effective at increasing clinicians’ perceived capability to implement COPD CARE. A survey was administered one week after Academy participation and a semi-structured interview conducted eight to 12 months later. Descriptive statistics were calculated for quantitative items and thematic analysis was used to analyze open-ended items. Results Thirty-six clinicians from 13 VA medical centers (VAMCs) participated in the Academy in 2020 and 2021 and 264 front-line clinicians completed COPD CARE training. Adoption of the Academy was indicated by high rates of Academy completion (97%), session attendance (90%), and high utilization of Academy resources. Clinicians reported the Academy to be acceptable and appropriate as an implementation package and clinicians from 92% of VAMCs reported long-term utilization of Academy resources. Effectiveness of the Academy was represented by clinicians’ significant increases (p < 0.05) in their capability to complete ten implementation tasks after Academy participation. Conclusions This evaluation found that the use of implementation facilitation paired with additional strategies seemed to demonstrate positive implementation outcomes across all RE-AIM domains and identified areas for potential improvement. Future assessments are needed to explore post-academy resources that would help VAMCs to strategize localized approaches to overcome barriers.

This evaluation explores the impact of a virtual implementation package, COPD Coordinated Access to Reduce Exacerbations (CARE) Academy (Academy), designed to scale a COPD care bundle in the Veterans' Health Administration (VA). The VA is the largest integrated health care system in the United States with 171 VA Medical Centers (VAMCs) with unique processes, cultures, priorities, and geographic barriers that can make scaling best practices di cult.
Initial design of the Academy began in 2018 with the development of a clinical training program that was re ned and tested across two VAMCs (16). The program was found to have a positive impact on clinician con dence and interprofessional collaboration, however clinicians reported they needed additional guidance and resources to overcome logistical barriers to implementing COPD CARE (17). Furthermore, a national implementation team, including experts in pharmacy and COPD management, recognized that different implementation strategies were needed to scale COPD CARE more rapidly.
To address these obstacles and promote effective service reach, the national implementation team developed a more comprehensive implementation package over a 12-month period with support from VA experts in Dissemination & Implementation (D&I) science (18). The Academy is comprised of four components, each of which is based on implementation strategies (19) and incorporate implementation facilitation (Fig. 1). Implementation Facilitation (IF) is an interactive approach to addressing implementation challenges through forming supportive relationships (20). The Academy provides IF through cohort-based learning by convening participants from multiple VAMCs working to implement the care bundle. Furthermore, external facilitators are included to promote discussion and shared problem solving.
Virtual discussions, guided implementation resources, informatics support and clinical training support were integrated within the IF model ( Fig. 1). [Insert Fig. 1] Evaluation conceptual framework We applied the RE-AIM conceptual framework (21,22) to measure the impact of the Academy. RE-AIM emphasizes translating evidencebased interventions into practice while also considering contextual factors that may facilitate or impede implementation of an intervention.
The framework de nes 5 domains and associated measures for examining an intervention's reach (R), effectiveness (E), adoption (A), implementation (I) and maintenance (M). RE-AIM was selected as a guiding framework to assess the Academy impact due to its inclusion of implementation outcomes in addition to effectiveness outcomes (21).
The speci c aims of this evaluation were to assess the impact of the Academy's implementation strategies on implementation outcomes (e.g., reach, adoption, implementation, and maintenance) and, ultimately the extent to which they were effective at increasing clinicians' self-e cacy to implement COPD CARE. Future evaluations are planned to examine more distal outcomes, such as implementation of COPD CARE best practices and improvements in Veteran care.

Design
This quality improvement evaluation utilized a mixed-methods approach to obtain retrospective feedback from clinicians about their perceptions of the Academy and its impact on their implementation of COPD CARE. This evaluation was determined not to meet the federal de nition of research and quali ed for a quality improvement exemption.

Setting and Sample
The Academy was implemented in two cohorts in Fall 2020 and Spring 2021. Cohort one involved ve Midwestern VAMCs and Cohort two involved eight VAMCs from the West and East coasts and the Southwest. Two approaches were used to identify the VAMCs: 1) the Academy was promoted through a national VA website known for promoting promising practices to VA leaders, and 2) program developers strategically engaged with Clinical Pharmacy Executives across the VA to identify VAMCs with a strong interest in the program. VAMCs that agreed to participate were instructed to complete a pre-implementation workbook, which involved identifying implementation team members, including an implementation lead (IL), responsible for guiding the overall process at their site and a clinician lead (CL) responsible for conducting the clinician training.

Data sources
Three primary data sources were used for this evaluation: 1) administrative records, 2) a survey, and 3) a semi-structured interview (Table 1). [Insert Table 1] Interviews conducted 8-12 months after completing the 5-week Academy Survey One week after completing the Academy, clinicians were invited to complete a 34-item Qualtrics survey to obtain their immediate feedback about the Academy and their perceptions of its impact on their capability to successfully accomplish implementation tasks. Six domains were addressed: 1) Academy participation, 2) perceptions of the Academy, 3) perceived capability to complete implementation tasks before and after Academy participation, 4) barriers to implementation, 5) opportunities for improvement, and 6) plans for implementation or adaptations to the COPD CARE service.

Semi-Structured Interviews
Eight to 12 months after Academy participation, implementation teams participated in a semi-structured interview to understand their experiences applying Academy resources within their VAMC. The interview guide consisted of questions related to three domains: 1) utilization of the Academy and perceptions of Academy content, 2) acceptability of Academy content delivery, and 3) experiences with COPD CARE implementation and adaptation. Eight of the 38 questions used a 7-point Likert scale asking clinicians to rate their agreement with a statement (i.e., 1 = very strongly agree, to 7 = very strongly disagree). The interviews lasted 60-75 minutes, involved up to four implementation team members, and were conducted over Zoom by a trained pharmacy intern. Interviews were recorded, auto-transcribed, and reviewed for accuracy.

Measures
Descriptions and data sources for each RE-AIM domain are detailed in Table 2. [Insert Table 2] • Survey items assessing perceived capability completing implementation tasks before and after Academy participation.
o Scale of 1 = "not at all capable," 5 = "moderately capable," and 10 = "highly capable" Adoption • Proportion of clinicians that attended Academy sessions, used Academy resources, and completed the Academy • Administrative program records of Academy completion, • Survey items assessing session attendance and use of resources.
• Semi-structured interview informed understanding of participants' reasons for participating in the Academy Implementation • Clinicians' perceptions of the acceptability and appropriateness of the Academy as an implementation package • Acceptability conceptualized as satisfaction with the Academy content and delivery approach • Appropriateness of the Academy = clinician's perceived t, usefulness, and practicality of the Academy with their VAMC (25).
• Survey items assessing Academy content and delivery approach, and relevance and usefulness of the Academy.

Data analysis
Descriptive statistics were calculated for all quantitative survey items and thematic analysis was used to summarize the qualitative openended items. To assess changes in self-reported capability, the non-parametric Wilcoxon signed-rank test was used for the 10 Likert scale items. No adjustments for repeated testing were made and an alpha level of 0.05 was used. IBM SPSS Statistics (Version 28) (23) was used for the statistical analysis. For structured interview data, descriptive statistics were calculated to summarize the eight Likert scale items and frequencies were calculated for the dichotomous yes/no items. An independent evaluator with no direct a liation to the VA conducted a thematic analysis of the open-ended items separately, and consensus of nal themes were agreed on through group discussion with the project lead. The thematic analysis was conducted using NVivo (24). Initially an inductive approach was taken using open coding. Survey and interview ndings are presented within the RE-AIM framework to identify indicators of the RE-AIM effectiveness and implementation outcomes for each domain. We used the SQUIRE 2.0 reporting guidelines when writing this paper (25).

RESULTS
The results are presented for each RE-AIM domain.

Reach
Thirty-six clinicians from 13 VAMCs participated in the Academy in Fall 2020 and Spring 2021. Across all 13 sites, 264 front-line clinicians completed the COPD CARE clinician training. This group included 130 pharmacists, 117 nurses and 17 other front-line clinicians (e.g., respiratory therapists).

Effectiveness
Sixteen clinicians from the 13 VAMCs responded to the survey. Thirteen (81%) respondents were Clinical Pharmacist Practitioners and three (19%) reported other professions (i.e., inpatient care med-surg Tele Nurse). Clinicians reported signi cant increases in their capability to complete implementation efforts after participation in the Academy across ten items representing implementation tasks (p < 0.05) ( Table 3). [Insert Table 3] Adoption Thirty-ve (97%) clinicians completed the Academy and over 90% of clinicians responding to the survey reported complete or nearly complete attendance at all ve of the Academy weekly discussions. Clinicians from 12 (92%) VAMCs participated in the semi-structured interview. Interviewed clinicians reported high utilization of Academy resources, with the workbooks being used by clinicians at all 12 (100%) VAMCs, followed by 11 (92%) VAMCs using the live virtual debrief meetings and the COPD CARE resources available through a shared network drive. Interview responses indicated that clinicians at three-fourths or more of VAMCs reported using the Academy weekly emails (83%), weekly YouTube videos (75%), and monthly post-Academy follow-up meetings (75%).

Implementation Acceptability
Clinicians' perceptions of the Academy content and delivery approach suggest they were satis ed with these aspects of the Academy (26).
Interview ndings indicate that, clinicians at all 12 (100%) sites viewed the Academy content to be complete and covering critical aspects of implementing the COPD CARE service and clinicians at 83% of VAMCs reported the approach to delivering Academy content was effective.
Clinicians reported that they valued the team-based support aspect of the Academy including the opportunities to be part of the virtual discussions and a learning collaborative. Survey results indicated that nearly all clinicians (94%) found that learning from colleagues at other VAMCs during the Academy and attending the weekly live sessions (81%), were some of the most valuable aspects of the Academy.
The interviews corroborated the survey ndings.  Table 4] However, clinicians also reported the Academy lacked su cient content in certain areas (e.g., exploring spirometry in greater depth, additional resources to describe the COPD CARE referral process, additional informatics support) ( Table 4).

Appropriateness of COPD CARE Academy
Step-by-step approach was appropriate […] we have a twice a year, education day and so our fall one was…I think it was half the day that was dedicated to COPD CARE so we all sat through the modules to get there and watch things together... (IL13) Long-term use of Academy resources […] I think that the handout provided and all the videos were really well done. I think most of us here we're kind of unfamiliar with COPD. And then, and then after going through it, again, I think the training was just was really good and the handouts, we still use we reference now has been a couple of months… (CL14) Bene t of monthly post-Academy meetings I mean some of the information that's been ongoing discussion in the community working group has been more helpful, because a lot of the questions my clinical pharmacists have were just like those higher level nuanced you know speci c case questions like, some of the things we've been talking about as far as like steroids deprescribing or concomitant asthma diagnosis. (IL9)

Appropriateness
Clinicians at nearly 70% of VAMCs perceived the Academy to be critically important and clinicians at 75% of VAMCs reported having the necessary tools and resources to implement COPD CARE after Academy participation. Clinicians found the external IF approach to be appropriate. Participants felt supported, motivated, and encouraged by the support from the national facilitator (Table 4). However, some clinicians reported challenges to Academy participation; describing ways in which the Academy may not have been perceived as suitable or practical as an implementation package for those VAMCs. For example, one clinician found that the weekly Academy topics were not always aligned with where their VAMC was in the implementation process (Table 4).

Maintenance
Clinicians from 92% of responding VAMCs reported long-term utilization of Academy resources and clinicians from 75% of VAMCs reported participating in the post-Academy meetings with other participants. The Academy had a lasting effect on sites, which was re ected in VAMC's integration of the Academy into the site organizational structure. Several clinicians reported that their implementation teams continued to have regular communication after the Academy (Table 4). For many, the Academy served as a lasting resource. Clinicians reported long-term use of the training materials and resources months after the Academy, suggesting its lasting effect and value (Table 4).

DISCUSSION
Guided by the RE-AIM framework we evaluated the Academy's impact on implementation outcomes and on increasing clinicians' perceived capability to implement COPD CARE. The use of IF as the overarching approach paired with additional strategies seemed to demonstrate positive outcomes across all RE-AIM domains. In this evaluation we found that the fully-virtual, cohort-based IF approach was successful at implementing COPD CARE at a large number of VAMCs simultaneously. This approach eliminates geographic and cost barriers to participation, increasing the reach of the Academy.
Interview ndings suggested clinicians were committed to Academy participation and had a high degree of resource utilization, indicating successful adoption. A majority of clinicians were satis ed with the content and delivery approach and viewed the Academy as a useful and practical approach to implementing COPD CARE, which are indicative of successful implementation. These positive RE-AIM implementation outcomes likely contributed to the effectiveness of the Academy at increasing clinicians' perceptions of their capability to implement COPD CARE. As clinicians embraced the Academy and participated in the virtual discussions with other VAMCs, they felt supported and their perceived capability to be successful at implementing COPD CARE increased. The linkage between the Academy's strategies and increasing perceived capability or self-e cacy is supported by Bandura's Social Cognitive Theory (27).
Findings from this evaluation are consistent with other studies that assessed the use of IF as a strategy to promote adoption of pharmacist-driven care bundles. One study used IF in a collaborative care intervention program to support the delivery of patient-centered clinical pain management services (28). Another study similarly applied IF strategies to promote the adoption of a treatment engagement intervention for homeless Veterans with a history of substance abuse (29). Lastly, a third study utilized IF to assess the implementation of primary care mental health integration care models (30). This assessment is the rst to the authors knowledge that integrates IF as an approach to enhance care transitions for COPD, with formation of new clinical teams to intentionally promote internal and external facilitation.
Despite the overwhelmingly positive outcomes in our evaluation, some clinicians identi ed challenges to participating in the Academy and highlighted content or resources they perceived to be lacking, such as additional training and informatics support to improve patient referrals. Opportunities exist to explore this barrier and enhance the COPD CARE referral process.
A limitation of this evaluation is its sole focus on the front-line clinician perspective and not other stakeholders' (e.g., leadership, clinic managers) perspectives. However, some contextual factors that may affect implementation, such as the clinicians' perceptions of leadership support, were explored. There may be other contextual factors at these VAMCs which posed barriers to successful engagement with the Academy and ultimately compromised COPD CARE implementation. Notably, these efforts to disseminate COPD CARE were made during a global pandemic, clinicians were not provided with additional salary support or protected time for their implementation efforts, and national facilitation of the program relied heavily on part-time employment from pharmacy student interns.

CONCLUSIONS
Through this evaluation we documented the positive impact the Academy had on implementation outcomes and enhancements to clinician perceptions of capability to implement successfully. We also identi ed potential areas of improvement for the Academy as an implementation package to support scale-up of COPD CARE. These lessons learned are important to inform future Academy improvements as it is rolled out to additional VAMCs. This evaluation adds to the growing evidence-base supporting the efforts to scale COPD CARE. It Authors' contributions ECP supervised the project, data collection, and analysis, as well as drafted and revised the paper. MAM conducted the quantitative and qualitative analysis and was a major contributor in writing the manuscript. JTK, SDB and ZZ collected quantitative and qualitative data and contributed to the writing and revisions of the paper. ECP, JTK, SDB, ZZ, SW and MSM were involved in the implementation of the intervention. RHG provided dissemination and implementation science expertise to the evaluation and revised the paper. NJ provided expertise in qualitative analysis to the data analysis and presentation of ndings and contributed to revisions of the paper. MAC, AMG, JAS, and CS provided critical feedback on the conceptualization of the analysis and presentation of the ndings and contributed to revisions of the paper. SW, MSM, and HO provided guidance throughout the implementation and evaluation and contributed to revisions of the paper. All authors read and approved the nal manuscript. Figure 1 Core Components of the Academy

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download. PortillocompletedSQUIREchecklist612023.docx